Analyzing the NYC Subway Dataset¶
Table of Contents¶
Section 0. References¶
Section 1. Statistical Test¶
1.1 Which statistical test did you use to analyze the NYC subway data? Did you use a one-tail or a two-tail P value? What is the null hypothesis? What is your p-critical value?¶
The test that we choose to analyze the NYC subway data is a Mann-Whitney U-test for a two tailed test. We analyze two conditions.The ridership during rainy days and the ridership during non-rainy days. The null hypothesis would be that a randomly selected value from the population with the larger mean rank which is the ridership in rainy days(see 1.3 answer below) is equal to a randomly selected value from the other population with the lower mean which is the ridership in non rainy days. And the alternative hypothesis would be that a randomly selected value from the population with the larger mean rank which is the ridership in rainy days is greater than a randomly selected value from the other population with the lower mean which is the ridership in non rainy days. The p-value for a two tailed t-test is twice the p-value output from the Mann-Whitney U Test. In our case, the p-value output is 0.024999912793489721 and the double is close to 0.05.
1.2 Why is this statistical test applicable to the dataset? In particular, consider the assumptions that the test is making about the distribution of ridership in the two samples.¶
Our samples are not normally distributed. Thus, we cannot use the welch's t-test for normally distributed independent samples.
1.3 What results did you get from this statistical test? These should include the following numerical values: p-values, as well as the means for each of the two samples under test.¶
do_rain_mean, no_rain_mean, U, p = (1105.4463767458733, 1090.278780151855, 1924409167.0, 0.024999912793489721)
1.4 What is the significance and interpretation of these results?¶
Our p-critical value is a bit lower than 0.05. Thus we assume that our test is statistically significant. The mean for rainy days is larger than the mean for non rainy days. So we reject the null hypothesis, meaning that a randomly selected value from the population with the larger mean rank which is the ridership in rainy days is greater than a randomly selected value from the other population with the lower mean which is the ridership in non rainy days.
Section 2. Linear Regression¶
2.1 What approach did you use to compute the coefficients theta and produce prediction for ENTRIESn_hourly in your regression model?¶
OLS using Statsmodels
2.2 What features (input variables) did you use in your model? Did you use any dummy variables as part of your features?¶
I used 'rain', 'precipi', 'Hour', 'meantempi'and 'fog' as input variables, 'UNIT' as dummy variable and 'ENTRIESn_hourly' as an output(values).
2.3 Why did you select these features in your model? We are looking for specific reasons that lead you to believe that the selected features will contribute to the predictive power of your model.¶
I supposed that these features would have a higher theta level. This could be tested with gradient descent.
2.4 What are the parameters (also known as "coefficients" or "weights") of the non-dummy features in your linear regression model?¶
The coefficients were the following 'rain' = -4.7007 'precipi' = -30.5838 'Hour' = 57.9852 'meantempi' = -12.3691 'fog' = 225.8609 constant = 1222.5636
2.5 What is your model’s R^2 (coefficients of determination) value?¶
R^2 = 0.47924770782
2.6 What does this R^2 value mean for the goodness of fit for your regression model? Do you think this linear model to predict ridership is appropriate for this dataset, given this R^2 value?¶
In order to be more accurate and evaluate the effectiveness of our model, we should calculate the coefficient of determination R^2. The closer this value is to 1, the better our model. In our case the R^2 is high enough to assume that we have a valuable model. Later, by calculating the residual frequency plot: {(turnstile_weather['ENTRIESn_hourly'] - predictions).plot(kind = 'hist', bins = 70) plt.axis([-12000, 12000, 0, 5000])} one can observe that the overall pattern of the residuals is similar to the bell-shaped pattern observed when plotting a histogram of normally distributed data. This gives us with proof that our assumptions are reasonable and our choice of model is appropriate. Nevertheless, the histogram of the residuals has long tails, which suggests that there are some very large residuals a reason to question our linear regression model.
Section 3. Visualization¶
3.1 One visualization should contain two histograms: one of ENTRIESn_hourly for rainy days and one of ENTRIESn_hourly for non-rainy days.¶
3.2 One visualization can be more freeform. You should feel free to implement something that we discussed in class (e.g., scatter plots, line plots) or attempt to implement something more advanced if you'd like. Some suggestions are:¶
- Ridership by time-of-day¶
- Ridership by day-of-week¶
%matplotlib inline
import numpy as np
import pandas
import matplotlib.pyplot as plt
import csv
turnstile_weather = pandas.read_csv('C:/Users/oikonomakisa/Desktop/turnstile_data_master_with_weather.csv')
#def entries_histogram(turnstile_weather):
plt.figure()
no_rain = turnstile_weather['ENTRIESn_hourly'][turnstile_weather['rain'] == 0]
do_rain = turnstile_weather['ENTRIESn_hourly'][turnstile_weather['rain'] == 1]
no_rain.hist(bins = 150, stacked=True, label = 'No Rain')
do_rain.hist(bins = 150, stacked=True, label = 'Rain')
plt.xlabel('ENTRIESn_hourly')
plt.ylabel('Frequency')
plt.title('Histogram of ENTRIESn_hourly')
plt.legend(loc='upper right')
plt.axis([0, 6000, 0, 45000])
# return plt
Fifure 1 output¶
From the above output, one can observe that the frequency of ridership at non rainy day is higher in low entries hourly. Which means that the train stations are emptier when in sunny days compared to rainy days.
%matplotlib inline
from pandas import *
from ggplot import *
turnstile_weather = pandas.read_csv('C:/Users/oikonomakisa/Desktop/turnstile_data_master_with_weather.csv')
pandas.options.mode.chained_assignment = None
turnstile_weather['weekday'] = pandas.to_datetime(turnstile_weather['DATEn']).apply(lambda x: x.strftime('%w'))
total = turnstile_weather.groupby(['weekday'], as_index=False)['ENTRIESn_hourly'].sum()
label_list = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
plot = ggplot(total, aes('weekday', 'ENTRIESn_hourly')) + geom_bar(stat = 'identity', color = 'blue') + ggtitle('Total Entries per Day') + scale_x_discrete(labels = label_list) + xlab('Day') + ylab('Rides')
plot
Fifure 2 output¶
One can observe that the ridership in weekends is lower than in weekdays.
Section 4. Conclusion¶
4.1 From your analysis and interpretation of the data, do more people ride the NYC subway when it is raining or when it is not raining?¶
From the anlysis and interpretation of the data, one can observe that the ridership in rainy days is a bit greater than the ridership in non rainy days.
4.2 What analyses lead you to this conclusion? You should use results from both your statistical tests and your linear regression to support your analysis.¶
We rejected the null hypothesis, using the Mann-Whitney U test, by observing the means and the p-value for a two-tailed test. Since the p-value is lower than the alpha level of 0.05, we ended up to say that we have a statistically significant conclusion.
Section 5. Reflection¶
5.1 Please discuss potential shortcomings of the methods of your analysis, including:¶
- Dataset,¶
- Analysis, such as the linear regression model or statistical test.¶
The residual normality testing in question 2.6 above, examined the fact that the histogram of the residuals has long tails, which suggests that there are some very large residuals a reason to question our linear regression model. Moreover, because there are many variables included in the dataset that might be very closely related, such as minimum, mean and maximum temperature, it may be difficult to disentangle the effects of such similar features and we may run the risk of problems with collinearity, which can cause some linear regression algorithms to give incorrect results. Lastly, I don't think that the model covers a long enough time span in order to make a more reliable prediction.